Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

materialize scannode slot for case 'select count* from select xxxx fr… #39

Merged
merged 1 commit into from
Aug 21, 2017
Merged

materialize scannode slot for case 'select count* from select xxxx fr… #39

merged 1 commit into from
Aug 21, 2017

Conversation

chenhao7253886
Copy link
Contributor

…om xxxx'

@chenhao7253886 chenhao7253886 merged commit f80aa73 into apache:master Aug 21, 2017
stdpain pushed a commit to stdpain/incubator-doris that referenced this pull request Jul 5, 2021
@yangzhg @HappenLee
remove time zone from data_time (apache#39)
07f61b7
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jul 15, 2021
@yangzhg @HappenLee
remove time zone from data_time (apache#39)
2b990e3
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Sep 7, 2021
@yangzhg @HappenLee
remove time zone from data_time (apache#39)
d6a0c3e
@dataroaring dataroaring mentioned this pull request Mar 30, 2022
Open
3 tasks
@deardeng deardeng mentioned this pull request May 31, 2022
Open
3 tasks
@SWJTU-ZhangLei SWJTU-ZhangLei mentioned this pull request Jun 10, 2022
Closed
3 tasks
platoneko added a commit to platoneko/doris that referenced this pull request Jun 20, 2022
@platoneko
Squashed commit of the following:
22c838b 
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date:   Mon Jun 20 15:37:21 2022 +0800

    temporarily avoid src/io conflicts with doris/master

commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date:   Fri Jun 17 20:59:41 2022 +0800

    (cold_on_s3) Report remote data size in tablets (apache#195)

    * be report tablet remote size

    * update TTabletStat

    * fe report tablet remote data size

commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date:   Fri Jun 17 16:06:23 2022 +0800

    (cold_on_s3) support modify partition add storage policy

commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date:   Fri Jun 17 10:24:14 2022 +0800

    (cold_on_s3) support alter table add storage policy

commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 16:01:02 2022 +0800

    (cold_on_s3) directly upload rowset exceeded ttl (apache#186)

    * directly upload rowset exceed ttl

    * clang-format

commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 11:39:08 2022 +0800

    (cold_on_s3) Implement and rewrite some filesystem methods (apache#183)

    * remove temporary test code

    * no need to  send fs to LocalFileWriter ctor

    * rewrite some fs methods

    * fix opened fd leak

commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date:   Tue Jun 14 18:07:52 2022 +0800

    (cold_on_s3) fix log printing (apache#176)

commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date:   Tue Jun 14 17:02:48 2022 +0800

    (cold_on_s3) fix rowset write timestamp bug (apache#175)

commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date:   Tue Jun 14 16:14:56 2022 +0800

    (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)

commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date:   Tue Jun 14 11:17:23 2022 +0800

    (cold_on_s3) fix compile error and thread pool core

commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date:   Mon Jun 13 19:15:37 2022 +0800

    (cold_on_s3) Revert FilePathDesc (apache#136)

    * change cooldown pick rowset strategy; add/modify some log

    * revert FilePathDesc

    * return unsupport error to avoid undefined behavior

    * patch to reduce path copy

    * fix be ut

commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:36 2022 +0800

    (cold_on_s3) Add ut for s3 read (apache#147)

commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:21 2022 +0800

    (cold_on_s3) Add metrics for file system operation (apache#146)

    * Add metrics for file system operation

    * fix status code

commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date:   Mon Jun 13 12:53:10 2022 +0800

    (cold_on_s3) support show proc backends return remote used capacity

commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date:   Fri Jun 10 16:08:09 2022 +0800

    (cold_on_s3) support storage policy use default policy

commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date:   Thu Jun 9 10:40:46 2022 +0800

    (cold_on_s3) create table support different partition use their own storage policy

commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date:   Fri Jun 10 22:28:40 2022 +0800

    (cold_on_s3) Implement some fs methods (apache#163)

    * implement some fs methods

    * add more log info

commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date:   Thu Jun 9 23:12:58 2022 +0800

    [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)

    apache#9986

commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date:   Thu Jun 9 16:17:06 2022 +0800

    [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)

commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date:   Wed Jun 8 16:09:35 2022 +0800

     (cold_on_s3) Optimize cooldown policy (apache#113)

    * Optimize cooldown policy

    * fix ut

    * remove useless log

    * add ut

    * add ut

    * rename var

    * fix typo and ut

commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date:   Wed Jun 8 16:02:41 2022 +0800

    provide quick_cooldown switch for regression test (apache#132)

commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date:   Tue Jun 7 18:05:01 2022 +0800

    (cold_on_s3) fix multiple fe synchronization resource storage policy exit

commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date:   Thu Jun 2 11:38:46 2022 +0800

    (cold_on_s3) fix move bug

commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date:   Thu Jun 2 11:24:16 2022 +0800

    (cold_on_s3) fix clang report '_exec_env' is not used

commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date:   Thu Jun 2 00:05:42 2022 +0800

    (cold_on_s3) reformat be code to make ci happy

commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date:   Wed Jun 1 23:44:56 2022 +0800

    (cold_on_s3) fix be ut

commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date:   Wed Jun 1 22:36:37 2022 +0800

    (cold_on_s3) fix fe compile error

commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date:   Thu May 26 11:15:31 2022 +0800

    [Enhancement](cold_on_s3) refresh fs map (apache#67)

    * refresh fs map

    * fetch cooldown ttl from storage policy

commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date:   Tue May 24 21:28:06 2022 +0800

    (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)

    * time from the beginning of upload

    * compact to appropriate state before upload

    * only do compaction on local rowsets

commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date:   Tue May 24 19:22:25 2022 +0800

    (cold_on_s3) use noncopy iostream for s3 get object (apache#54)

commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date:   Tue May 24 18:42:09 2022 +0800

    (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)

commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date:   Tue May 24 12:58:43 2022 +0800

    (cold_on_s3) support storage policy change fix thrift merge redefine

commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date:   Tue May 24 11:45:13 2022 +0800

    (cold_on_s3) support storage policy change and create table add storage policy name (apache#39)

    * [feature](cold-hot)support storage policy change and create table add storage policy name

    * support storage policy change and create table add storage policy name, check resource policy exist when create table.

    * when create table use storage policy check it exist

    * support storage policy change and create table add storage policy name and  when create table use storage policy check it exist

    * support storage policy change fix some bug.

    * fix some code style

commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date:   Sat May 21 07:37:43 2022 -0700

    (cold_on_s3) support storage policy (apache#28)

    * [feature](cold-hot)support storage policy

    * fix review

    * fix review v1

commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date:   Fri May 20 16:22:36 2022 +0800

    (cold_on_s3) Batch upload to improve upload speed (apache#38)

    * test upload duration

    * use s3 transfer

    * batch upload

    * some fix in s3 filesystem

commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date:   Wed May 18 15:08:40 2022 +0800

    (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)

    * Implement periodic uploading of rowsets to remote storage

    * add queue size check

    * add test config for cooldwon

    * modify read path

commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date:   Tue May 17 17:22:19 2022 +0800

    (cold_on_s3) fe distinguish drop partition (apache#26)

commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date:   Mon May 16 21:02:51 2022 -0700

    (cold_on_s3) support clone task don't download file in s3 (apache#15)

    * apache#12 support clone task don't download file in s3

    * apache#12 add clone ut

    * fix code format

    * fix commit apache#12

commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date:   Tue May 17 12:01:32 2022 +0800

    [Enhancement](cold_on_s3) FE distinguish drop table or drop replica;  disk capacity distinguish local or remote (apache#24)

    * disk capacity differentiates between local and remote

    * fe drop table

commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date:   Mon May 16 11:55:42 2022 +0800

    [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)

    * update is_local logical

    * remove remote rowsets when drop table/partition

commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date:   Mon May 16 11:17:30 2022 +0800

    (cold_on_s3) Implement reading rowset data from remote storage (apache#11)

    * adjust io stack

    * support s3

    * refine code

    * fix ut

    * remove useless code

    * rename rblock

    * fix err msg

commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date:   Wed May 11 16:29:54 2022 +0800

    (cold_on_s3) Implement tablet cooldown (apache#10)

    * tablet cooldown

    * add global_local_filesystem to keep compatible

    * test tablet cooldown

    * revert engine_storage_migration_task_test.cpp

commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date:   Sun May 1 12:06:41 2022 +0800

    (cold_on_s3) Rowset write path using fs (apache#7)

    * replace WritableBlock with FileWriter

    * local file system

    * set fs when load dir

    * fix be ut

commit 3585fe6
Author: Xinyi Zou <[email protected]>
Date:   Tue Jun 7 11:31:49 2022 +0800

    [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)

    * fix tracker 0602

    * fix format

commit 429a41d
Author: Xinyi Zou <[email protected]>
Date:   Tue May 31 19:12:42 2022 +0800

    [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate

commit 89d3a36
Author: Pxl <[email protected]>
Date:   Thu May 26 16:51:01 2022 +0800

    [Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko added a commit to platoneko/doris that referenced this pull request Jun 20, 2022
@platoneko
Squashed commit of the following: (#1)
1886ba2 
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date:   Mon Jun 20 15:37:21 2022 +0800

    temporarily avoid src/io conflicts with doris/master

commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date:   Fri Jun 17 20:59:41 2022 +0800

    (cold_on_s3) Report remote data size in tablets (apache#195)

    * be report tablet remote size

    * update TTabletStat

    * fe report tablet remote data size

commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date:   Fri Jun 17 16:06:23 2022 +0800

    (cold_on_s3) support modify partition add storage policy

commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date:   Fri Jun 17 10:24:14 2022 +0800

    (cold_on_s3) support alter table add storage policy

commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 16:01:02 2022 +0800

    (cold_on_s3) directly upload rowset exceeded ttl (apache#186)

    * directly upload rowset exceed ttl

    * clang-format

commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 11:39:08 2022 +0800

    (cold_on_s3) Implement and rewrite some filesystem methods (apache#183)

    * remove temporary test code

    * no need to  send fs to LocalFileWriter ctor

    * rewrite some fs methods

    * fix opened fd leak

commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date:   Tue Jun 14 18:07:52 2022 +0800

    (cold_on_s3) fix log printing (apache#176)

commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date:   Tue Jun 14 17:02:48 2022 +0800

    (cold_on_s3) fix rowset write timestamp bug (apache#175)

commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date:   Tue Jun 14 16:14:56 2022 +0800

    (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)

commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date:   Tue Jun 14 11:17:23 2022 +0800

    (cold_on_s3) fix compile error and thread pool core

commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date:   Mon Jun 13 19:15:37 2022 +0800

    (cold_on_s3) Revert FilePathDesc (apache#136)

    * change cooldown pick rowset strategy; add/modify some log

    * revert FilePathDesc

    * return unsupport error to avoid undefined behavior

    * patch to reduce path copy

    * fix be ut

commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:36 2022 +0800

    (cold_on_s3) Add ut for s3 read (apache#147)

commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:21 2022 +0800

    (cold_on_s3) Add metrics for file system operation (apache#146)

    * Add metrics for file system operation

    * fix status code

commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date:   Mon Jun 13 12:53:10 2022 +0800

    (cold_on_s3) support show proc backends return remote used capacity

commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date:   Fri Jun 10 16:08:09 2022 +0800

    (cold_on_s3) support storage policy use default policy

commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date:   Thu Jun 9 10:40:46 2022 +0800

    (cold_on_s3) create table support different partition use their own storage policy

commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date:   Fri Jun 10 22:28:40 2022 +0800

    (cold_on_s3) Implement some fs methods (apache#163)

    * implement some fs methods

    * add more log info

commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date:   Thu Jun 9 23:12:58 2022 +0800

    [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)

    apache#9986

commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date:   Thu Jun 9 16:17:06 2022 +0800

    [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)

commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date:   Wed Jun 8 16:09:35 2022 +0800

     (cold_on_s3) Optimize cooldown policy (apache#113)

    * Optimize cooldown policy

    * fix ut

    * remove useless log

    * add ut

    * add ut

    * rename var

    * fix typo and ut

commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date:   Wed Jun 8 16:02:41 2022 +0800

    provide quick_cooldown switch for regression test (apache#132)

commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date:   Tue Jun 7 18:05:01 2022 +0800

    (cold_on_s3) fix multiple fe synchronization resource storage policy exit

commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date:   Thu Jun 2 11:38:46 2022 +0800

    (cold_on_s3) fix move bug

commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date:   Thu Jun 2 11:24:16 2022 +0800

    (cold_on_s3) fix clang report '_exec_env' is not used

commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date:   Thu Jun 2 00:05:42 2022 +0800

    (cold_on_s3) reformat be code to make ci happy

commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date:   Wed Jun 1 23:44:56 2022 +0800

    (cold_on_s3) fix be ut

commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date:   Wed Jun 1 22:36:37 2022 +0800

    (cold_on_s3) fix fe compile error

commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date:   Thu May 26 11:15:31 2022 +0800

    [Enhancement](cold_on_s3) refresh fs map (apache#67)

    * refresh fs map

    * fetch cooldown ttl from storage policy

commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date:   Tue May 24 21:28:06 2022 +0800

    (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)

    * time from the beginning of upload

    * compact to appropriate state before upload

    * only do compaction on local rowsets

commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date:   Tue May 24 19:22:25 2022 +0800

    (cold_on_s3) use noncopy iostream for s3 get object (apache#54)

commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date:   Tue May 24 18:42:09 2022 +0800

    (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)

commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date:   Tue May 24 12:58:43 2022 +0800

    (cold_on_s3) support storage policy change fix thrift merge redefine

commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date:   Tue May 24 11:45:13 2022 +0800

    (cold_on_s3) support storage policy change and create table add storage policy name (apache#39)

    * [feature](cold-hot)support storage policy change and create table add storage policy name

    * support storage policy change and create table add storage policy name, check resource policy exist when create table.

    * when create table use storage policy check it exist

    * support storage policy change and create table add storage policy name and  when create table use storage policy check it exist

    * support storage policy change fix some bug.

    * fix some code style

commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date:   Sat May 21 07:37:43 2022 -0700

    (cold_on_s3) support storage policy (apache#28)

    * [feature](cold-hot)support storage policy

    * fix review

    * fix review v1

commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date:   Fri May 20 16:22:36 2022 +0800

    (cold_on_s3) Batch upload to improve upload speed (apache#38)

    * test upload duration

    * use s3 transfer

    * batch upload

    * some fix in s3 filesystem

commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date:   Wed May 18 15:08:40 2022 +0800

    (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)

    * Implement periodic uploading of rowsets to remote storage

    * add queue size check

    * add test config for cooldwon

    * modify read path

commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date:   Tue May 17 17:22:19 2022 +0800

    (cold_on_s3) fe distinguish drop partition (apache#26)

commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date:   Mon May 16 21:02:51 2022 -0700

    (cold_on_s3) support clone task don't download file in s3 (apache#15)

    * apache#12 support clone task don't download file in s3

    * apache#12 add clone ut

    * fix code format

    * fix commit apache#12

commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date:   Tue May 17 12:01:32 2022 +0800

    [Enhancement](cold_on_s3) FE distinguish drop table or drop replica;  disk capacity distinguish local or remote (apache#24)

    * disk capacity differentiates between local and remote

    * fe drop table

commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date:   Mon May 16 11:55:42 2022 +0800

    [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)

    * update is_local logical

    * remove remote rowsets when drop table/partition

commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date:   Mon May 16 11:17:30 2022 +0800

    (cold_on_s3) Implement reading rowset data from remote storage (apache#11)

    * adjust io stack

    * support s3

    * refine code

    * fix ut

    * remove useless code

    * rename rblock

    * fix err msg

commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date:   Wed May 11 16:29:54 2022 +0800

    (cold_on_s3) Implement tablet cooldown (apache#10)

    * tablet cooldown

    * add global_local_filesystem to keep compatible

    * test tablet cooldown

    * revert engine_storage_migration_task_test.cpp

commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date:   Sun May 1 12:06:41 2022 +0800

    (cold_on_s3) Rowset write path using fs (apache#7)

    * replace WritableBlock with FileWriter

    * local file system

    * set fs when load dir

    * fix be ut

commit 3585fe6
Author: Xinyi Zou <[email protected]>
Date:   Tue Jun 7 11:31:49 2022 +0800

    [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)

    * fix tracker 0602

    * fix format

commit 429a41d
Author: Xinyi Zou <[email protected]>
Date:   Tue May 31 19:12:42 2022 +0800

    [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate

commit 89d3a36
Author: Pxl <[email protected]>
Date:   Thu May 26 16:51:01 2022 +0800

    [Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko added a commit to platoneko/doris that referenced this pull request Jun 20, 2022
@platoneko
Squashed commit of the following: (#2)
c7cfe01 
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date:   Mon Jun 20 15:37:21 2022 +0800

    temporarily avoid src/io conflicts with doris/master

commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date:   Fri Jun 17 20:59:41 2022 +0800

    (cold_on_s3) Report remote data size in tablets (apache#195)

    * be report tablet remote size

    * update TTabletStat

    * fe report tablet remote data size

commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date:   Fri Jun 17 16:06:23 2022 +0800

    (cold_on_s3) support modify partition add storage policy

commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date:   Fri Jun 17 10:24:14 2022 +0800

    (cold_on_s3) support alter table add storage policy

commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 16:01:02 2022 +0800

    (cold_on_s3) directly upload rowset exceeded ttl (apache#186)

    * directly upload rowset exceed ttl

    * clang-format

commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 11:39:08 2022 +0800

    (cold_on_s3) Implement and rewrite some filesystem methods (apache#183)

    * remove temporary test code

    * no need to  send fs to LocalFileWriter ctor

    * rewrite some fs methods

    * fix opened fd leak

commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date:   Tue Jun 14 18:07:52 2022 +0800

    (cold_on_s3) fix log printing (apache#176)

commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date:   Tue Jun 14 17:02:48 2022 +0800

    (cold_on_s3) fix rowset write timestamp bug (apache#175)

commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date:   Tue Jun 14 16:14:56 2022 +0800

    (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)

commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date:   Tue Jun 14 11:17:23 2022 +0800

    (cold_on_s3) fix compile error and thread pool core

commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date:   Mon Jun 13 19:15:37 2022 +0800

    (cold_on_s3) Revert FilePathDesc (apache#136)

    * change cooldown pick rowset strategy; add/modify some log

    * revert FilePathDesc

    * return unsupport error to avoid undefined behavior

    * patch to reduce path copy

    * fix be ut

commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:36 2022 +0800

    (cold_on_s3) Add ut for s3 read (apache#147)

commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:21 2022 +0800

    (cold_on_s3) Add metrics for file system operation (apache#146)

    * Add metrics for file system operation

    * fix status code

commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date:   Mon Jun 13 12:53:10 2022 +0800

    (cold_on_s3) support show proc backends return remote used capacity

commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date:   Fri Jun 10 16:08:09 2022 +0800

    (cold_on_s3) support storage policy use default policy

commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date:   Thu Jun 9 10:40:46 2022 +0800

    (cold_on_s3) create table support different partition use their own storage policy

commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date:   Fri Jun 10 22:28:40 2022 +0800

    (cold_on_s3) Implement some fs methods (apache#163)

    * implement some fs methods

    * add more log info

commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date:   Thu Jun 9 23:12:58 2022 +0800

    [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)

    apache#9986

commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date:   Thu Jun 9 16:17:06 2022 +0800

    [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)

commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date:   Wed Jun 8 16:09:35 2022 +0800

     (cold_on_s3) Optimize cooldown policy (apache#113)

    * Optimize cooldown policy

    * fix ut

    * remove useless log

    * add ut

    * add ut

    * rename var

    * fix typo and ut

commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date:   Wed Jun 8 16:02:41 2022 +0800

    provide quick_cooldown switch for regression test (apache#132)

commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date:   Tue Jun 7 18:05:01 2022 +0800

    (cold_on_s3) fix multiple fe synchronization resource storage policy exit

commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date:   Thu Jun 2 11:38:46 2022 +0800

    (cold_on_s3) fix move bug

commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date:   Thu Jun 2 11:24:16 2022 +0800

    (cold_on_s3) fix clang report '_exec_env' is not used

commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date:   Thu Jun 2 00:05:42 2022 +0800

    (cold_on_s3) reformat be code to make ci happy

commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date:   Wed Jun 1 23:44:56 2022 +0800

    (cold_on_s3) fix be ut

commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date:   Wed Jun 1 22:36:37 2022 +0800

    (cold_on_s3) fix fe compile error

commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date:   Thu May 26 11:15:31 2022 +0800

    [Enhancement](cold_on_s3) refresh fs map (apache#67)

    * refresh fs map

    * fetch cooldown ttl from storage policy

commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date:   Tue May 24 21:28:06 2022 +0800

    (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)

    * time from the beginning of upload

    * compact to appropriate state before upload

    * only do compaction on local rowsets

commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date:   Tue May 24 19:22:25 2022 +0800

    (cold_on_s3) use noncopy iostream for s3 get object (apache#54)

commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date:   Tue May 24 18:42:09 2022 +0800

    (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)

commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date:   Tue May 24 12:58:43 2022 +0800

    (cold_on_s3) support storage policy change fix thrift merge redefine

commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date:   Tue May 24 11:45:13 2022 +0800

    (cold_on_s3) support storage policy change and create table add storage policy name (apache#39)

    * [feature](cold-hot)support storage policy change and create table add storage policy name

    * support storage policy change and create table add storage policy name, check resource policy exist when create table.

    * when create table use storage policy check it exist

    * support storage policy change and create table add storage policy name and  when create table use storage policy check it exist

    * support storage policy change fix some bug.

    * fix some code style

commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date:   Sat May 21 07:37:43 2022 -0700

    (cold_on_s3) support storage policy (apache#28)

    * [feature](cold-hot)support storage policy

    * fix review

    * fix review v1

commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date:   Fri May 20 16:22:36 2022 +0800

    (cold_on_s3) Batch upload to improve upload speed (apache#38)

    * test upload duration

    * use s3 transfer

    * batch upload

    * some fix in s3 filesystem

commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date:   Wed May 18 15:08:40 2022 +0800

    (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)

    * Implement periodic uploading of rowsets to remote storage

    * add queue size check

    * add test config for cooldwon

    * modify read path

commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date:   Tue May 17 17:22:19 2022 +0800

    (cold_on_s3) fe distinguish drop partition (apache#26)

commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date:   Mon May 16 21:02:51 2022 -0700

    (cold_on_s3) support clone task don't download file in s3 (apache#15)

    * apache#12 support clone task don't download file in s3

    * apache#12 add clone ut

    * fix code format

    * fix commit apache#12

commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date:   Tue May 17 12:01:32 2022 +0800

    [Enhancement](cold_on_s3) FE distinguish drop table or drop replica;  disk capacity distinguish local or remote (apache#24)

    * disk capacity differentiates between local and remote

    * fe drop table

commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date:   Mon May 16 11:55:42 2022 +0800

    [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)

    * update is_local logical

    * remove remote rowsets when drop table/partition

commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date:   Mon May 16 11:17:30 2022 +0800

    (cold_on_s3) Implement reading rowset data from remote storage (apache#11)

    * adjust io stack

    * support s3

    * refine code

    * fix ut

    * remove useless code

    * rename rblock

    * fix err msg

commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date:   Wed May 11 16:29:54 2022 +0800

    (cold_on_s3) Implement tablet cooldown (apache#10)

    * tablet cooldown

    * add global_local_filesystem to keep compatible

    * test tablet cooldown

    * revert engine_storage_migration_task_test.cpp

commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date:   Sun May 1 12:06:41 2022 +0800

    (cold_on_s3) Rowset write path using fs (apache#7)

    * replace WritableBlock with FileWriter

    * local file system

    * set fs when load dir

    * fix be ut

    Co-authored-by: Luwei <[email protected]>
    Co-authored-by: deardeng <[email protected]>
platoneko added a commit to platoneko/doris that referenced this pull request Jun 20, 2022
@platoneko
Co-authored-by: Luwei <[email protected]>
9dbdde5 
    Co-authored-by: deardeng <[email protected]>

Squashed commit of the following:

commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date:   Mon Jun 20 15:37:21 2022 +0800

    temporarily avoid src/io conflicts with doris/master

commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date:   Fri Jun 17 20:59:41 2022 +0800

    (cold_on_s3) Report remote data size in tablets (apache#195)

    * be report tablet remote size

    * update TTabletStat

    * fe report tablet remote data size

commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date:   Fri Jun 17 16:06:23 2022 +0800

    (cold_on_s3) support modify partition add storage policy

commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date:   Fri Jun 17 10:24:14 2022 +0800

    (cold_on_s3) support alter table add storage policy

commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 16:01:02 2022 +0800

    (cold_on_s3) directly upload rowset exceeded ttl (apache#186)

    * directly upload rowset exceed ttl

    * clang-format

commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 11:39:08 2022 +0800

    (cold_on_s3) Implement and rewrite some filesystem methods (apache#183)

    * remove temporary test code

    * no need to  send fs to LocalFileWriter ctor

    * rewrite some fs methods

    * fix opened fd leak

commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date:   Tue Jun 14 18:07:52 2022 +0800

    (cold_on_s3) fix log printing (apache#176)

commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date:   Tue Jun 14 17:02:48 2022 +0800

    (cold_on_s3) fix rowset write timestamp bug (apache#175)

commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date:   Tue Jun 14 16:14:56 2022 +0800

    (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)

commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date:   Tue Jun 14 11:17:23 2022 +0800

    (cold_on_s3) fix compile error and thread pool core

commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date:   Mon Jun 13 19:15:37 2022 +0800

    (cold_on_s3) Revert FilePathDesc (apache#136)

    * change cooldown pick rowset strategy; add/modify some log

    * revert FilePathDesc

    * return unsupport error to avoid undefined behavior

    * patch to reduce path copy

    * fix be ut

commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:36 2022 +0800

    (cold_on_s3) Add ut for s3 read (apache#147)

commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:21 2022 +0800

    (cold_on_s3) Add metrics for file system operation (apache#146)

    * Add metrics for file system operation

    * fix status code

commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date:   Mon Jun 13 12:53:10 2022 +0800

    (cold_on_s3) support show proc backends return remote used capacity

commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date:   Fri Jun 10 16:08:09 2022 +0800

    (cold_on_s3) support storage policy use default policy

commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date:   Thu Jun 9 10:40:46 2022 +0800

    (cold_on_s3) create table support different partition use their own storage policy

commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date:   Fri Jun 10 22:28:40 2022 +0800

    (cold_on_s3) Implement some fs methods (apache#163)

    * implement some fs methods

    * add more log info

commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date:   Thu Jun 9 23:12:58 2022 +0800

    [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)

    apache#9986

commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date:   Thu Jun 9 16:17:06 2022 +0800

    [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)

commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date:   Wed Jun 8 16:09:35 2022 +0800

     (cold_on_s3) Optimize cooldown policy (apache#113)

    * Optimize cooldown policy

    * fix ut

    * remove useless log

    * add ut

    * add ut

    * rename var

    * fix typo and ut

commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date:   Wed Jun 8 16:02:41 2022 +0800

    provide quick_cooldown switch for regression test (apache#132)

commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date:   Tue Jun 7 18:05:01 2022 +0800

    (cold_on_s3) fix multiple fe synchronization resource storage policy exit

commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date:   Thu Jun 2 11:38:46 2022 +0800

    (cold_on_s3) fix move bug

commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date:   Thu Jun 2 11:24:16 2022 +0800

    (cold_on_s3) fix clang report '_exec_env' is not used

commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date:   Thu Jun 2 00:05:42 2022 +0800

    (cold_on_s3) reformat be code to make ci happy

commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date:   Wed Jun 1 23:44:56 2022 +0800

    (cold_on_s3) fix be ut

commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date:   Wed Jun 1 22:36:37 2022 +0800

    (cold_on_s3) fix fe compile error

commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date:   Thu May 26 11:15:31 2022 +0800

    [Enhancement](cold_on_s3) refresh fs map (apache#67)

    * refresh fs map

    * fetch cooldown ttl from storage policy

commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date:   Tue May 24 21:28:06 2022 +0800

    (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)

    * time from the beginning of upload

    * compact to appropriate state before upload

    * only do compaction on local rowsets

commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date:   Tue May 24 19:22:25 2022 +0800

    (cold_on_s3) use noncopy iostream for s3 get object (apache#54)

commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date:   Tue May 24 18:42:09 2022 +0800

    (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)

commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date:   Tue May 24 12:58:43 2022 +0800

    (cold_on_s3) support storage policy change fix thrift merge redefine

commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date:   Tue May 24 11:45:13 2022 +0800

    (cold_on_s3) support storage policy change and create table add storage policy name (apache#39)

    * [feature](cold-hot)support storage policy change and create table add storage policy name

    * support storage policy change and create table add storage policy name, check resource policy exist when create table.

    * when create table use storage policy check it exist

    * support storage policy change and create table add storage policy name and  when create table use storage policy check it exist

    * support storage policy change fix some bug.

    * fix some code style

commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date:   Sat May 21 07:37:43 2022 -0700

    (cold_on_s3) support storage policy (apache#28)

    * [feature](cold-hot)support storage policy

    * fix review

    * fix review v1

commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date:   Fri May 20 16:22:36 2022 +0800

    (cold_on_s3) Batch upload to improve upload speed (apache#38)

    * test upload duration

    * use s3 transfer

    * batch upload

    * some fix in s3 filesystem

commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date:   Wed May 18 15:08:40 2022 +0800

    (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)

    * Implement periodic uploading of rowsets to remote storage

    * add queue size check

    * add test config for cooldwon

    * modify read path

commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date:   Tue May 17 17:22:19 2022 +0800

    (cold_on_s3) fe distinguish drop partition (apache#26)

commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date:   Mon May 16 21:02:51 2022 -0700

    (cold_on_s3) support clone task don't download file in s3 (apache#15)

    * apache#12 support clone task don't download file in s3

    * apache#12 add clone ut

    * fix code format

    * fix commit apache#12

commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date:   Tue May 17 12:01:32 2022 +0800

    [Enhancement](cold_on_s3) FE distinguish drop table or drop replica;  disk capacity distinguish local or remote (apache#24)

    * disk capacity differentiates between local and remote

    * fe drop table

commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date:   Mon May 16 11:55:42 2022 +0800

    [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)

    * update is_local logical

    * remove remote rowsets when drop table/partition

commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date:   Mon May 16 11:17:30 2022 +0800

    (cold_on_s3) Implement reading rowset data from remote storage (apache#11)

    * adjust io stack

    * support s3

    * refine code

    * fix ut

    * remove useless code

    * rename rblock

    * fix err msg

commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date:   Wed May 11 16:29:54 2022 +0800

    (cold_on_s3) Implement tablet cooldown (apache#10)

    * tablet cooldown

    * add global_local_filesystem to keep compatible

    * test tablet cooldown

    * revert engine_storage_migration_task_test.cpp

commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date:   Sun May 1 12:06:41 2022 +0800

    (cold_on_s3) Rowset write path using fs (apache#7)

    * replace WritableBlock with FileWriter

    * local file system

    * set fs when load dir

    * fix be ut

commit 3585fe6
Author: Xinyi Zou <[email protected]>
Date:   Tue Jun 7 11:31:49 2022 +0800

    [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)

    * fix tracker 0602

    * fix format

commit 429a41d
Author: Xinyi Zou <[email protected]>
Date:   Tue May 31 19:12:42 2022 +0800

    [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate

commit 89d3a36
Author: Pxl <[email protected]>
Date:   Thu May 26 16:51:01 2022 +0800

    [Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko added a commit to platoneko/doris that referenced this pull request Jun 20, 2022
@platoneko @luwei16 @deardeng
Squashed commit of the following:
4491627 
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date:   Mon Jun 20 15:37:21 2022 +0800

    temporarily avoid src/io conflicts with doris/master

commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date:   Fri Jun 17 20:59:41 2022 +0800

    (cold_on_s3) Report remote data size in tablets (apache#195)

    * be report tablet remote size

    * update TTabletStat

    * fe report tablet remote data size

commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date:   Fri Jun 17 16:06:23 2022 +0800

    (cold_on_s3) support modify partition add storage policy

commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date:   Fri Jun 17 10:24:14 2022 +0800

    (cold_on_s3) support alter table add storage policy

commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 16:01:02 2022 +0800

    (cold_on_s3) directly upload rowset exceeded ttl (apache#186)

    * directly upload rowset exceed ttl

    * clang-format

commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 11:39:08 2022 +0800

    (cold_on_s3) Implement and rewrite some filesystem methods (apache#183)

    * remove temporary test code

    * no need to  send fs to LocalFileWriter ctor

    * rewrite some fs methods

    * fix opened fd leak

commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date:   Tue Jun 14 18:07:52 2022 +0800

    (cold_on_s3) fix log printing (apache#176)

commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date:   Tue Jun 14 17:02:48 2022 +0800

    (cold_on_s3) fix rowset write timestamp bug (apache#175)

commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date:   Tue Jun 14 16:14:56 2022 +0800

    (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)

commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date:   Tue Jun 14 11:17:23 2022 +0800

    (cold_on_s3) fix compile error and thread pool core

commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date:   Mon Jun 13 19:15:37 2022 +0800

    (cold_on_s3) Revert FilePathDesc (apache#136)

    * change cooldown pick rowset strategy; add/modify some log

    * revert FilePathDesc

    * return unsupport error to avoid undefined behavior

    * patch to reduce path copy

    * fix be ut

commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:36 2022 +0800

    (cold_on_s3) Add ut for s3 read (apache#147)

commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:21 2022 +0800

    (cold_on_s3) Add metrics for file system operation (apache#146)

    * Add metrics for file system operation

    * fix status code

commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date:   Mon Jun 13 12:53:10 2022 +0800

    (cold_on_s3) support show proc backends return remote used capacity

commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date:   Fri Jun 10 16:08:09 2022 +0800

    (cold_on_s3) support storage policy use default policy

commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date:   Thu Jun 9 10:40:46 2022 +0800

    (cold_on_s3) create table support different partition use their own storage policy

commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date:   Fri Jun 10 22:28:40 2022 +0800

    (cold_on_s3) Implement some fs methods (apache#163)

    * implement some fs methods

    * add more log info

commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date:   Thu Jun 9 23:12:58 2022 +0800

    [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)

    apache#9986

commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date:   Thu Jun 9 16:17:06 2022 +0800

    [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)

commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date:   Wed Jun 8 16:09:35 2022 +0800

     (cold_on_s3) Optimize cooldown policy (apache#113)

    * Optimize cooldown policy

    * fix ut

    * remove useless log

    * add ut

    * add ut

    * rename var

    * fix typo and ut

commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date:   Wed Jun 8 16:02:41 2022 +0800

    provide quick_cooldown switch for regression test (apache#132)

commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date:   Tue Jun 7 18:05:01 2022 +0800

    (cold_on_s3) fix multiple fe synchronization resource storage policy exit

commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date:   Thu Jun 2 11:38:46 2022 +0800

    (cold_on_s3) fix move bug

commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date:   Thu Jun 2 11:24:16 2022 +0800

    (cold_on_s3) fix clang report '_exec_env' is not used

commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date:   Thu Jun 2 00:05:42 2022 +0800

    (cold_on_s3) reformat be code to make ci happy

commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date:   Wed Jun 1 23:44:56 2022 +0800

    (cold_on_s3) fix be ut

commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date:   Wed Jun 1 22:36:37 2022 +0800

    (cold_on_s3) fix fe compile error

commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date:   Thu May 26 11:15:31 2022 +0800

    [Enhancement](cold_on_s3) refresh fs map (apache#67)

    * refresh fs map

    * fetch cooldown ttl from storage policy

commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date:   Tue May 24 21:28:06 2022 +0800

    (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)

    * time from the beginning of upload

    * compact to appropriate state before upload

    * only do compaction on local rowsets

commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date:   Tue May 24 19:22:25 2022 +0800

    (cold_on_s3) use noncopy iostream for s3 get object (apache#54)

commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date:   Tue May 24 18:42:09 2022 +0800

    (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)

commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date:   Tue May 24 12:58:43 2022 +0800

    (cold_on_s3) support storage policy change fix thrift merge redefine

commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date:   Tue May 24 11:45:13 2022 +0800

    (cold_on_s3) support storage policy change and create table add storage policy name (apache#39)

    * [feature](cold-hot)support storage policy change and create table add storage policy name

    * support storage policy change and create table add storage policy name, check resource policy exist when create table.

    * when create table use storage policy check it exist

    * support storage policy change and create table add storage policy name and  when create table use storage policy check it exist

    * support storage policy change fix some bug.

    * fix some code style

commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date:   Sat May 21 07:37:43 2022 -0700

    (cold_on_s3) support storage policy (apache#28)

    * [feature](cold-hot)support storage policy

    * fix review

    * fix review v1

commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date:   Fri May 20 16:22:36 2022 +0800

    (cold_on_s3) Batch upload to improve upload speed (apache#38)

    * test upload duration

    * use s3 transfer

    * batch upload

    * some fix in s3 filesystem

commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date:   Wed May 18 15:08:40 2022 +0800

    (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)

    * Implement periodic uploading of rowsets to remote storage

    * add queue size check

    * add test config for cooldwon

    * modify read path

commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date:   Tue May 17 17:22:19 2022 +0800

    (cold_on_s3) fe distinguish drop partition (apache#26)

commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date:   Mon May 16 21:02:51 2022 -0700

    (cold_on_s3) support clone task don't download file in s3 (apache#15)

    * apache#12 support clone task don't download file in s3

    * apache#12 add clone ut

    * fix code format

    * fix commit apache#12

commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date:   Tue May 17 12:01:32 2022 +0800

    [Enhancement](cold_on_s3) FE distinguish drop table or drop replica;  disk capacity distinguish local or remote (apache#24)

    * disk capacity differentiates between local and remote

    * fe drop table

commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date:   Mon May 16 11:55:42 2022 +0800

    [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)

    * update is_local logical

    * remove remote rowsets when drop table/partition

commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date:   Mon May 16 11:17:30 2022 +0800

    (cold_on_s3) Implement reading rowset data from remote storage (apache#11)

    * adjust io stack

    * support s3

    * refine code

    * fix ut

    * remove useless code

    * rename rblock

    * fix err msg

commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date:   Wed May 11 16:29:54 2022 +0800

    (cold_on_s3) Implement tablet cooldown (apache#10)

    * tablet cooldown

    * add global_local_filesystem to keep compatible

    * test tablet cooldown

    * revert engine_storage_migration_task_test.cpp

commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date:   Sun May 1 12:06:41 2022 +0800

    (cold_on_s3) Rowset write path using fs (apache#7)

    * replace WritableBlock with FileWriter

    * local file system

    * set fs when load dir

    * fix be ut

commit 3585fe6
Author: Xinyi Zou <[email protected]>
Date:   Tue Jun 7 11:31:49 2022 +0800

    [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)

    * fix tracker 0602

    * fix format

commit 429a41d
Author: Xinyi Zou <[email protected]>
Date:   Tue May 31 19:12:42 2022 +0800

    [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate

commit 89d3a36
Author: Pxl <[email protected]>
Date:   Thu May 26 16:51:01 2022 +0800

    [Bug] [Vectorized] add padding when load char type data (apache#9734)

Co-authored-by: Luwei <[email protected]>
Co-authored-by: deardeng <[email protected]>
platoneko added a commit to platoneko/doris that referenced this pull request Jul 6, 2022
@platoneko @luwei16 @deardeng
Squashed commit of the following(beyond FE):
d2da08b 
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date:   Mon Jun 20 15:37:21 2022 +0800

    temporarily avoid src/io conflicts with doris/master

commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date:   Fri Jun 17 20:59:41 2022 +0800

    (cold_on_s3) Report remote data size in tablets (apache#195)

    * be report tablet remote size

    * update TTabletStat

    * fe report tablet remote data size

commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date:   Fri Jun 17 16:06:23 2022 +0800

    (cold_on_s3) support modify partition add storage policy

commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date:   Fri Jun 17 10:24:14 2022 +0800

    (cold_on_s3) support alter table add storage policy

commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 16:01:02 2022 +0800

    (cold_on_s3) directly upload rowset exceeded ttl (apache#186)

    * directly upload rowset exceed ttl

    * clang-format

commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 11:39:08 2022 +0800

    (cold_on_s3) Implement and rewrite some filesystem methods (apache#183)

    * remove temporary test code

    * no need to  send fs to LocalFileWriter ctor

    * rewrite some fs methods

    * fix opened fd leak

commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date:   Tue Jun 14 18:07:52 2022 +0800

    (cold_on_s3) fix log printing (apache#176)

commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date:   Tue Jun 14 17:02:48 2022 +0800

    (cold_on_s3) fix rowset write timestamp bug (apache#175)

commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date:   Tue Jun 14 16:14:56 2022 +0800

    (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)

commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date:   Tue Jun 14 11:17:23 2022 +0800

    (cold_on_s3) fix compile error and thread pool core

commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date:   Mon Jun 13 19:15:37 2022 +0800

    (cold_on_s3) Revert FilePathDesc (apache#136)

    * change cooldown pick rowset strategy; add/modify some log

    * revert FilePathDesc

    * return unsupport error to avoid undefined behavior

    * patch to reduce path copy

    * fix be ut

commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:36 2022 +0800

    (cold_on_s3) Add ut for s3 read (apache#147)

commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:21 2022 +0800

    (cold_on_s3) Add metrics for file system operation (apache#146)

    * Add metrics for file system operation

    * fix status code

commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date:   Mon Jun 13 12:53:10 2022 +0800

    (cold_on_s3) support show proc backends return remote used capacity

commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date:   Fri Jun 10 16:08:09 2022 +0800

    (cold_on_s3) support storage policy use default policy

commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date:   Thu Jun 9 10:40:46 2022 +0800

    (cold_on_s3) create table support different partition use their own storage policy

commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date:   Fri Jun 10 22:28:40 2022 +0800

    (cold_on_s3) Implement some fs methods (apache#163)

    * implement some fs methods

    * add more log info

commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date:   Thu Jun 9 23:12:58 2022 +0800

    [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)

    apache#9986

commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date:   Thu Jun 9 16:17:06 2022 +0800

    [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)

commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date:   Wed Jun 8 16:09:35 2022 +0800

     (cold_on_s3) Optimize cooldown policy (apache#113)

    * Optimize cooldown policy

    * fix ut

    * remove useless log

    * add ut

    * add ut

    * rename var

    * fix typo and ut

commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date:   Wed Jun 8 16:02:41 2022 +0800

    provide quick_cooldown switch for regression test (apache#132)

commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date:   Tue Jun 7 18:05:01 2022 +0800

    (cold_on_s3) fix multiple fe synchronization resource storage policy exit

commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date:   Thu Jun 2 11:38:46 2022 +0800

    (cold_on_s3) fix move bug

commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date:   Thu Jun 2 11:24:16 2022 +0800

    (cold_on_s3) fix clang report '_exec_env' is not used

commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date:   Thu Jun 2 00:05:42 2022 +0800

    (cold_on_s3) reformat be code to make ci happy

commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date:   Wed Jun 1 23:44:56 2022 +0800

    (cold_on_s3) fix be ut

commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date:   Wed Jun 1 22:36:37 2022 +0800

    (cold_on_s3) fix fe compile error

commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date:   Thu May 26 11:15:31 2022 +0800

    [Enhancement](cold_on_s3) refresh fs map (apache#67)

    * refresh fs map

    * fetch cooldown ttl from storage policy

commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date:   Tue May 24 21:28:06 2022 +0800

    (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)

    * time from the beginning of upload

    * compact to appropriate state before upload

    * only do compaction on local rowsets

commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date:   Tue May 24 19:22:25 2022 +0800

    (cold_on_s3) use noncopy iostream for s3 get object (apache#54)

commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date:   Tue May 24 18:42:09 2022 +0800

    (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)

commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date:   Tue May 24 12:58:43 2022 +0800

    (cold_on_s3) support storage policy change fix thrift merge redefine

commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date:   Tue May 24 11:45:13 2022 +0800

    (cold_on_s3) support storage policy change and create table add storage policy name (apache#39)

    * [feature](cold-hot)support storage policy change and create table add storage policy name

    * support storage policy change and create table add storage policy name, check resource policy exist when create table.

    * when create table use storage policy check it exist

    * support storage policy change and create table add storage policy name and  when create table use storage policy check it exist

    * support storage policy change fix some bug.

    * fix some code style

commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date:   Sat May 21 07:37:43 2022 -0700

    (cold_on_s3) support storage policy (apache#28)

    * [feature](cold-hot)support storage policy

    * fix review

    * fix review v1

commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date:   Fri May 20 16:22:36 2022 +0800

    (cold_on_s3) Batch upload to improve upload speed (apache#38)

    * test upload duration

    * use s3 transfer

    * batch upload

    * some fix in s3 filesystem

commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date:   Wed May 18 15:08:40 2022 +0800

    (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)

    * Implement periodic uploading of rowsets to remote storage

    * add queue size check

    * add test config for cooldwon

    * modify read path

commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date:   Tue May 17 17:22:19 2022 +0800

    (cold_on_s3) fe distinguish drop partition (apache#26)

commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date:   Mon May 16 21:02:51 2022 -0700

    (cold_on_s3) support clone task don't download file in s3 (apache#15)

    * apache#12 support clone task don't download file in s3

    * apache#12 add clone ut

    * fix code format

    * fix commit apache#12

commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date:   Tue May 17 12:01:32 2022 +0800

    [Enhancement](cold_on_s3) FE distinguish drop table or drop replica;  disk capacity distinguish local or remote (apache#24)

    * disk capacity differentiates between local and remote

    * fe drop table

commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date:   Mon May 16 11:55:42 2022 +0800

    [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)

    * update is_local logical

    * remove remote rowsets when drop table/partition

commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date:   Mon May 16 11:17:30 2022 +0800

    (cold_on_s3) Implement reading rowset data from remote storage (apache#11)

    * adjust io stack

    * support s3

    * refine code

    * fix ut

    * remove useless code

    * rename rblock

    * fix err msg

commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date:   Wed May 11 16:29:54 2022 +0800

    (cold_on_s3) Implement tablet cooldown (apache#10)

    * tablet cooldown

    * add global_local_filesystem to keep compatible

    * test tablet cooldown

    * revert engine_storage_migration_task_test.cpp

commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date:   Sun May 1 12:06:41 2022 +0800

    (cold_on_s3) Rowset write path using fs (apache#7)

    * replace WritableBlock with FileWriter

    * local file system

    * set fs when load dir

    * fix be ut

commit 3585fe6
Author: Xinyi Zou <[email protected]>
Date:   Tue Jun 7 11:31:49 2022 +0800

    [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)

    * fix tracker 0602

    * fix format

commit 429a41d
Author: Xinyi Zou <[email protected]>
Date:   Tue May 31 19:12:42 2022 +0800

    [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate

commit 89d3a36
Author: Pxl <[email protected]>
Date:   Thu May 26 16:51:01 2022 +0800

    [Bug] [Vectorized] add padding when load char type data (apache#9734)

Co-authored-by: Luwei <[email protected]>
Co-authored-by: deardeng <[email protected]>
platoneko added a commit to platoneko/doris that referenced this pull request Jul 6, 2022
@platoneko @luwei16 @deardeng
Squashed commit of the following(only BE):
24f3871 
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date:   Mon Jun 20 15:37:21 2022 +0800

    temporarily avoid src/io conflicts with doris/master

commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date:   Fri Jun 17 20:59:41 2022 +0800

    (cold_on_s3) Report remote data size in tablets (apache#195)

    * be report tablet remote size

    * update TTabletStat

    * fe report tablet remote data size

commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date:   Fri Jun 17 16:06:23 2022 +0800

    (cold_on_s3) support modify partition add storage policy

commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date:   Fri Jun 17 10:24:14 2022 +0800

    (cold_on_s3) support alter table add storage policy

commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 16:01:02 2022 +0800

    (cold_on_s3) directly upload rowset exceeded ttl (apache#186)

    * directly upload rowset exceed ttl

    * clang-format

commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 11:39:08 2022 +0800

    (cold_on_s3) Implement and rewrite some filesystem methods (apache#183)

    * remove temporary test code

    * no need to  send fs to LocalFileWriter ctor

    * rewrite some fs methods

    * fix opened fd leak

commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date:   Tue Jun 14 18:07:52 2022 +0800

    (cold_on_s3) fix log printing (apache#176)

commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date:   Tue Jun 14 17:02:48 2022 +0800

    (cold_on_s3) fix rowset write timestamp bug (apache#175)

commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date:   Tue Jun 14 16:14:56 2022 +0800

    (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)

commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date:   Tue Jun 14 11:17:23 2022 +0800

    (cold_on_s3) fix compile error and thread pool core

commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date:   Mon Jun 13 19:15:37 2022 +0800

    (cold_on_s3) Revert FilePathDesc (apache#136)

    * change cooldown pick rowset strategy; add/modify some log

    * revert FilePathDesc

    * return unsupport error to avoid undefined behavior

    * patch to reduce path copy

    * fix be ut

commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:36 2022 +0800

    (cold_on_s3) Add ut for s3 read (apache#147)

commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:21 2022 +0800

    (cold_on_s3) Add metrics for file system operation (apache#146)

    * Add metrics for file system operation

    * fix status code

commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date:   Mon Jun 13 12:53:10 2022 +0800

    (cold_on_s3) support show proc backends return remote used capacity

commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date:   Fri Jun 10 16:08:09 2022 +0800

    (cold_on_s3) support storage policy use default policy

commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date:   Thu Jun 9 10:40:46 2022 +0800

    (cold_on_s3) create table support different partition use their own storage policy

commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date:   Fri Jun 10 22:28:40 2022 +0800

    (cold_on_s3) Implement some fs methods (apache#163)

    * implement some fs methods

    * add more log info

commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date:   Thu Jun 9 23:12:58 2022 +0800

    [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)

    apache#9986

commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date:   Thu Jun 9 16:17:06 2022 +0800

    [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)

commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date:   Wed Jun 8 16:09:35 2022 +0800

     (cold_on_s3) Optimize cooldown policy (apache#113)

    * Optimize cooldown policy

    * fix ut

    * remove useless log

    * add ut

    * add ut

    * rename var

    * fix typo and ut

commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date:   Wed Jun 8 16:02:41 2022 +0800

    provide quick_cooldown switch for regression test (apache#132)

commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date:   Tue Jun 7 18:05:01 2022 +0800

    (cold_on_s3) fix multiple fe synchronization resource storage policy exit

commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date:   Thu Jun 2 11:38:46 2022 +0800

    (cold_on_s3) fix move bug

commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date:   Thu Jun 2 11:24:16 2022 +0800

    (cold_on_s3) fix clang report '_exec_env' is not used

commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date:   Thu Jun 2 00:05:42 2022 +0800

    (cold_on_s3) reformat be code to make ci happy

commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date:   Wed Jun 1 23:44:56 2022 +0800

    (cold_on_s3) fix be ut

commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date:   Wed Jun 1 22:36:37 2022 +0800

    (cold_on_s3) fix fe compile error

commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date:   Thu May 26 11:15:31 2022 +0800

    [Enhancement](cold_on_s3) refresh fs map (apache#67)

    * refresh fs map

    * fetch cooldown ttl from storage policy

commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date:   Tue May 24 21:28:06 2022 +0800

    (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)

    * time from the beginning of upload

    * compact to appropriate state before upload

    * only do compaction on local rowsets

commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date:   Tue May 24 19:22:25 2022 +0800

    (cold_on_s3) use noncopy iostream for s3 get object (apache#54)

commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date:   Tue May 24 18:42:09 2022 +0800

    (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)

commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date:   Tue May 24 12:58:43 2022 +0800

    (cold_on_s3) support storage policy change fix thrift merge redefine

commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date:   Tue May 24 11:45:13 2022 +0800

    (cold_on_s3) support storage policy change and create table add storage policy name (apache#39)

    * [feature](cold-hot)support storage policy change and create table add storage policy name

    * support storage policy change and create table add storage policy name, check resource policy exist when create table.

    * when create table use storage policy check it exist

    * support storage policy change and create table add storage policy name and  when create table use storage policy check it exist

    * support storage policy change fix some bug.

    * fix some code style

commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date:   Sat May 21 07:37:43 2022 -0700

    (cold_on_s3) support storage policy (apache#28)

    * [feature](cold-hot)support storage policy

    * fix review

    * fix review v1

commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date:   Fri May 20 16:22:36 2022 +0800

    (cold_on_s3) Batch upload to improve upload speed (apache#38)

    * test upload duration

    * use s3 transfer

    * batch upload

    * some fix in s3 filesystem

commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date:   Wed May 18 15:08:40 2022 +0800

    (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)

    * Implement periodic uploading of rowsets to remote storage

    * add queue size check

    * add test config for cooldwon

    * modify read path

commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date:   Tue May 17 17:22:19 2022 +0800

    (cold_on_s3) fe distinguish drop partition (apache#26)

commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date:   Mon May 16 21:02:51 2022 -0700

    (cold_on_s3) support clone task don't download file in s3 (apache#15)

    * apache#12 support clone task don't download file in s3

    * apache#12 add clone ut

    * fix code format

    * fix commit apache#12

commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date:   Tue May 17 12:01:32 2022 +0800

    [Enhancement](cold_on_s3) FE distinguish drop table or drop replica;  disk capacity distinguish local or remote (apache#24)

    * disk capacity differentiates between local and remote

    * fe drop table

commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date:   Mon May 16 11:55:42 2022 +0800

    [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)

    * update is_local logical

    * remove remote rowsets when drop table/partition

commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date:   Mon May 16 11:17:30 2022 +0800

    (cold_on_s3) Implement reading rowset data from remote storage (apache#11)

    * adjust io stack

    * support s3

    * refine code

    * fix ut

    * remove useless code

    * rename rblock

    * fix err msg

commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date:   Wed May 11 16:29:54 2022 +0800

    (cold_on_s3) Implement tablet cooldown (apache#10)

    * tablet cooldown

    * add global_local_filesystem to keep compatible

    * test tablet cooldown

    * revert engine_storage_migration_task_test.cpp

commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date:   Sun May 1 12:06:41 2022 +0800

    (cold_on_s3) Rowset write path using fs (apache#7)

    * replace WritableBlock with FileWriter

    * local file system

    * set fs when load dir

    * fix be ut

commit 3585fe6
Author: Xinyi Zou <[email protected]>
Date:   Tue Jun 7 11:31:49 2022 +0800

    [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)

    * fix tracker 0602

    * fix format

commit 429a41d
Author: Xinyi Zou <[email protected]>
Date:   Tue May 31 19:12:42 2022 +0800

    [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate

commit 89d3a36
Author: Pxl <[email protected]>
Date:   Thu May 26 16:51:01 2022 +0800

    [Bug] [Vectorized] add padding when load char type data (apache#9734)

Co-authored-by: Luwei <[email protected]>
Co-authored-by: deardeng <[email protected]>
platoneko added a commit to platoneko/doris that referenced this pull request Jul 6, 2022
@platoneko @luwei16 @deardeng
Squashed commit of the following(only BE):
7d13bb1 
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date:   Mon Jun 20 15:37:21 2022 +0800

    temporarily avoid src/io conflicts with doris/master

commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date:   Fri Jun 17 20:59:41 2022 +0800

    (cold_on_s3) Report remote data size in tablets (apache#195)

    * be report tablet remote size

    * update TTabletStat

    * fe report tablet remote data size

commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date:   Fri Jun 17 16:06:23 2022 +0800

    (cold_on_s3) support modify partition add storage policy

commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date:   Fri Jun 17 10:24:14 2022 +0800

    (cold_on_s3) support alter table add storage policy

commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 16:01:02 2022 +0800

    (cold_on_s3) directly upload rowset exceeded ttl (apache#186)

    * directly upload rowset exceed ttl

    * clang-format

commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 11:39:08 2022 +0800

    (cold_on_s3) Implement and rewrite some filesystem methods (apache#183)

    * remove temporary test code

    * no need to  send fs to LocalFileWriter ctor

    * rewrite some fs methods

    * fix opened fd leak

commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date:   Tue Jun 14 18:07:52 2022 +0800

    (cold_on_s3) fix log printing (apache#176)

commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date:   Tue Jun 14 17:02:48 2022 +0800

    (cold_on_s3) fix rowset write timestamp bug (apache#175)

commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date:   Tue Jun 14 16:14:56 2022 +0800

    (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)

commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date:   Tue Jun 14 11:17:23 2022 +0800

    (cold_on_s3) fix compile error and thread pool core

commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date:   Mon Jun 13 19:15:37 2022 +0800

    (cold_on_s3) Revert FilePathDesc (apache#136)

    * change cooldown pick rowset strategy; add/modify some log

    * revert FilePathDesc

    * return unsupport error to avoid undefined behavior

    * patch to reduce path copy

    * fix be ut

commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:36 2022 +0800

    (cold_on_s3) Add ut for s3 read (apache#147)

commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:21 2022 +0800

    (cold_on_s3) Add metrics for file system operation (apache#146)

    * Add metrics for file system operation

    * fix status code

commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date:   Mon Jun 13 12:53:10 2022 +0800

    (cold_on_s3) support show proc backends return remote used capacity

commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date:   Fri Jun 10 16:08:09 2022 +0800

    (cold_on_s3) support storage policy use default policy

commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date:   Thu Jun 9 10:40:46 2022 +0800

    (cold_on_s3) create table support different partition use their own storage policy

commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date:   Fri Jun 10 22:28:40 2022 +0800

    (cold_on_s3) Implement some fs methods (apache#163)

    * implement some fs methods

    * add more log info

commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date:   Thu Jun 9 23:12:58 2022 +0800

    [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)

    apache#9986

commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date:   Thu Jun 9 16:17:06 2022 +0800

    [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)

commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date:   Wed Jun 8 16:09:35 2022 +0800

     (cold_on_s3) Optimize cooldown policy (apache#113)

    * Optimize cooldown policy

    * fix ut

    * remove useless log

    * add ut

    * add ut

    * rename var

    * fix typo and ut

commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date:   Wed Jun 8 16:02:41 2022 +0800

    provide quick_cooldown switch for regression test (apache#132)

commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date:   Tue Jun 7 18:05:01 2022 +0800

    (cold_on_s3) fix multiple fe synchronization resource storage policy exit

commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date:   Thu Jun 2 11:38:46 2022 +0800

    (cold_on_s3) fix move bug

commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date:   Thu Jun 2 11:24:16 2022 +0800

    (cold_on_s3) fix clang report '_exec_env' is not used

commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date:   Thu Jun 2 00:05:42 2022 +0800

    (cold_on_s3) reformat be code to make ci happy

commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date:   Wed Jun 1 23:44:56 2022 +0800

    (cold_on_s3) fix be ut

commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date:   Wed Jun 1 22:36:37 2022 +0800

    (cold_on_s3) fix fe compile error

commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date:   Thu May 26 11:15:31 2022 +0800

    [Enhancement](cold_on_s3) refresh fs map (apache#67)

    * refresh fs map

    * fetch cooldown ttl from storage policy

commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date:   Tue May 24 21:28:06 2022 +0800

    (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)

    * time from the beginning of upload

    * compact to appropriate state before upload

    * only do compaction on local rowsets

commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date:   Tue May 24 19:22:25 2022 +0800

    (cold_on_s3) use noncopy iostream for s3 get object (apache#54)

commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date:   Tue May 24 18:42:09 2022 +0800

    (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)

commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date:   Tue May 24 12:58:43 2022 +0800

    (cold_on_s3) support storage policy change fix thrift merge redefine

commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date:   Tue May 24 11:45:13 2022 +0800

    (cold_on_s3) support storage policy change and create table add storage policy name (apache#39)

    * [feature](cold-hot)support storage policy change and create table add storage policy name

    * support storage policy change and create table add storage policy name, check resource policy exist when create table.

    * when create table use storage policy check it exist

    * support storage policy change and create table add storage policy name and  when create table use storage policy check it exist

    * support storage policy change fix some bug.

    * fix some code style

commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date:   Sat May 21 07:37:43 2022 -0700

    (cold_on_s3) support storage policy (apache#28)

    * [feature](cold-hot)support storage policy

    * fix review

    * fix review v1

commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date:   Fri May 20 16:22:36 2022 +0800

    (cold_on_s3) Batch upload to improve upload speed (apache#38)

    * test upload duration

    * use s3 transfer

    * batch upload

    * some fix in s3 filesystem

commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date:   Wed May 18 15:08:40 2022 +0800

    (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)

    * Implement periodic uploading of rowsets to remote storage

    * add queue size check

    * add test config for cooldwon

    * modify read path

commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date:   Tue May 17 17:22:19 2022 +0800

    (cold_on_s3) fe distinguish drop partition (apache#26)

commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date:   Mon May 16 21:02:51 2022 -0700

    (cold_on_s3) support clone task don't download file in s3 (apache#15)

    * apache#12 support clone task don't download file in s3

    * apache#12 add clone ut

    * fix code format

    * fix commit apache#12

commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date:   Tue May 17 12:01:32 2022 +0800

    [Enhancement](cold_on_s3) FE distinguish drop table or drop replica;  disk capacity distinguish local or remote (apache#24)

    * disk capacity differentiates between local and remote

    * fe drop table

commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date:   Mon May 16 11:55:42 2022 +0800

    [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)

    * update is_local logical

    * remove remote rowsets when drop table/partition

commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date:   Mon May 16 11:17:30 2022 +0800

    (cold_on_s3) Implement reading rowset data from remote storage (apache#11)

    * adjust io stack

    * support s3

    * refine code

    * fix ut

    * remove useless code

    * rename rblock

    * fix err msg

commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date:   Wed May 11 16:29:54 2022 +0800

    (cold_on_s3) Implement tablet cooldown (apache#10)

    * tablet cooldown

    * add global_local_filesystem to keep compatible

    * test tablet cooldown

    * revert engine_storage_migration_task_test.cpp

commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date:   Sun May 1 12:06:41 2022 +0800

    (cold_on_s3) Rowset write path using fs (apache#7)

    * replace WritableBlock with FileWriter

    * local file system

    * set fs when load dir

    * fix be ut

commit 3585fe6
Author: Xinyi Zou <[email protected]>
Date:   Tue Jun 7 11:31:49 2022 +0800

    [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)

    * fix tracker 0602

    * fix format

commit 429a41d
Author: Xinyi Zou <[email protected]>
Date:   Tue May 31 19:12:42 2022 +0800

    [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate

commit 89d3a36
Author: Pxl <[email protected]>
Date:   Thu May 26 16:51:01 2022 +0800

    [Bug] [Vectorized] add padding when load char type data (apache#9734)

Co-authored-by: Luwei <[email protected]>
Co-authored-by: deardeng <[email protected]>
platoneko added a commit to platoneko/doris that referenced this pull request Jul 7, 2022
@platoneko @luwei16 @deardeng
Squashed commit of the following(only BE):
61c2736 
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date:   Mon Jun 20 15:37:21 2022 +0800

    temporarily avoid src/io conflicts with doris/master

commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date:   Fri Jun 17 20:59:41 2022 +0800

    (cold_on_s3) Report remote data size in tablets (apache#195)

    * be report tablet remote size

    * update TTabletStat

    * fe report tablet remote data size

commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date:   Fri Jun 17 16:06:23 2022 +0800

    (cold_on_s3) support modify partition add storage policy

commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date:   Fri Jun 17 10:24:14 2022 +0800

    (cold_on_s3) support alter table add storage policy

commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 16:01:02 2022 +0800

    (cold_on_s3) directly upload rowset exceeded ttl (apache#186)

    * directly upload rowset exceed ttl

    * clang-format

commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date:   Thu Jun 16 11:39:08 2022 +0800

    (cold_on_s3) Implement and rewrite some filesystem methods (apache#183)

    * remove temporary test code

    * no need to  send fs to LocalFileWriter ctor

    * rewrite some fs methods

    * fix opened fd leak

commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date:   Tue Jun 14 18:07:52 2022 +0800

    (cold_on_s3) fix log printing (apache#176)

commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date:   Tue Jun 14 17:02:48 2022 +0800

    (cold_on_s3) fix rowset write timestamp bug (apache#175)

commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date:   Tue Jun 14 16:14:56 2022 +0800

    (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)

commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date:   Tue Jun 14 11:17:23 2022 +0800

    (cold_on_s3) fix compile error and thread pool core

commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date:   Mon Jun 13 19:15:37 2022 +0800

    (cold_on_s3) Revert FilePathDesc (apache#136)

    * change cooldown pick rowset strategy; add/modify some log

    * revert FilePathDesc

    * return unsupport error to avoid undefined behavior

    * patch to reduce path copy

    * fix be ut

commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:36 2022 +0800

    (cold_on_s3) Add ut for s3 read (apache#147)

commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date:   Mon Jun 13 16:07:21 2022 +0800

    (cold_on_s3) Add metrics for file system operation (apache#146)

    * Add metrics for file system operation

    * fix status code

commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date:   Mon Jun 13 12:53:10 2022 +0800

    (cold_on_s3) support show proc backends return remote used capacity

commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date:   Fri Jun 10 16:08:09 2022 +0800

    (cold_on_s3) support storage policy use default policy

commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date:   Thu Jun 9 10:40:46 2022 +0800

    (cold_on_s3) create table support different partition use their own storage policy

commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date:   Fri Jun 10 22:28:40 2022 +0800

    (cold_on_s3) Implement some fs methods (apache#163)

    * implement some fs methods

    * add more log info

commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date:   Thu Jun 9 23:12:58 2022 +0800

    [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)

    apache#9986

commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date:   Thu Jun 9 16:17:06 2022 +0800

    [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)

commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date:   Wed Jun 8 16:09:35 2022 +0800

     (cold_on_s3) Optimize cooldown policy (apache#113)

    * Optimize cooldown policy

    * fix ut

    * remove useless log

    * add ut

    * add ut

    * rename var

    * fix typo and ut

commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date:   Wed Jun 8 16:02:41 2022 +0800

    provide quick_cooldown switch for regression test (apache#132)

commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date:   Tue Jun 7 18:05:01 2022 +0800

    (cold_on_s3) fix multiple fe synchronization resource storage policy exit

commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date:   Thu Jun 2 11:38:46 2022 +0800

    (cold_on_s3) fix move bug

commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date:   Thu Jun 2 11:24:16 2022 +0800

    (cold_on_s3) fix clang report '_exec_env' is not used

commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date:   Thu Jun 2 00:05:42 2022 +0800

    (cold_on_s3) reformat be code to make ci happy

commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date:   Wed Jun 1 23:44:56 2022 +0800

    (cold_on_s3) fix be ut

commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date:   Wed Jun 1 22:36:37 2022 +0800

    (cold_on_s3) fix fe compile error

commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date:   Thu May 26 11:15:31 2022 +0800

    [Enhancement](cold_on_s3) refresh fs map (apache#67)

    * refresh fs map

    * fetch cooldown ttl from storage policy

commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date:   Tue May 24 21:28:06 2022 +0800

    (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)

    * time from the beginning of upload

    * compact to appropriate state before upload

    * only do compaction on local rowsets

commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date:   Tue May 24 19:22:25 2022 +0800

    (cold_on_s3) use noncopy iostream for s3 get object (apache#54)

commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date:   Tue May 24 18:42:09 2022 +0800

    (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)

commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date:   Tue May 24 12:58:43 2022 +0800

    (cold_on_s3) support storage policy change fix thrift merge redefine

commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date:   Tue May 24 11:45:13 2022 +0800

    (cold_on_s3) support storage policy change and create table add storage policy name (apache#39)

    * [feature](cold-hot)support storage policy change and create table add storage policy name

    * support storage policy change and create table add storage policy name, check resource policy exist when create table.

    * when create table use storage policy check it exist

    * support storage policy change and create table add storage policy name and  when create table use storage policy check it exist

    * support storage policy change fix some bug.

    * fix some code style

commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date:   Sat May 21 07:37:43 2022 -0700

    (cold_on_s3) support storage policy (apache#28)

    * [feature](cold-hot)support storage policy

    * fix review

    * fix review v1

commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date:   Fri May 20 16:22:36 2022 +0800

    (cold_on_s3) Batch upload to improve upload speed (apache#38)

    * test upload duration

    * use s3 transfer

    * batch upload

    * some fix in s3 filesystem

commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date:   Wed May 18 15:08:40 2022 +0800

    (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)

    * Implement periodic uploading of rowsets to remote storage

    * add queue size check

    * add test config for cooldwon

    * modify read path

commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date:   Tue May 17 17:22:19 2022 +0800

    (cold_on_s3) fe distinguish drop partition (apache#26)

commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date:   Mon May 16 21:02:51 2022 -0700

    (cold_on_s3) support clone task don't download file in s3 (apache#15)

    * apache#12 support clone task don't download file in s3

    * apache#12 add clone ut

    * fix code format

    * fix commit apache#12

commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date:   Tue May 17 12:01:32 2022 +0800

    [Enhancement](cold_on_s3) FE distinguish drop table or drop replica;  disk capacity distinguish local or remote (apache#24)

    * disk capacity differentiates between local and remote

    * fe drop table

commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date:   Mon May 16 11:55:42 2022 +0800

    [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)

    * update is_local logical

    * remove remote rowsets when drop table/partition

commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date:   Mon May 16 11:17:30 2022 +0800

    (cold_on_s3) Implement reading rowset data from remote storage (apache#11)

    * adjust io stack

    * support s3

    * refine code

    * fix ut

    * remove useless code

    * rename rblock

    * fix err msg

commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date:   Wed May 11 16:29:54 2022 +0800

    (cold_on_s3) Implement tablet cooldown (apache#10)

    * tablet cooldown

    * add global_local_filesystem to keep compatible

    * test tablet cooldown

    * revert engine_storage_migration_task_test.cpp

commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date:   Sun May 1 12:06:41 2022 +0800

    (cold_on_s3) Rowset write path using fs (apache#7)

    * replace WritableBlock with FileWriter

    * local file system

    * set fs when load dir

    * fix be ut

commit 3585fe6
Author: Xinyi Zou <[email protected]>
Date:   Tue Jun 7 11:31:49 2022 +0800

    [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)

    * fix tracker 0602

    * fix format

commit 429a41d
Author: Xinyi Zou <[email protected]>
Date:   Tue May 31 19:12:42 2022 +0800

    [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate

commit 89d3a36
Author: Pxl <[email protected]>
Date:   Thu May 26 16:51:01 2022 +0800

    [Bug] [Vectorized] add padding when load char type data (apache#9734)

Co-authored-by: Luwei <[email protected]>
Co-authored-by: deardeng <[email protected]>
@eldenmoon eldenmoon mentioned this pull request Jul 14, 2022
Closed
3 tasks
@zhangstar333 zhangstar333 mentioned this pull request Oct 14, 2022
Closed
3 tasks
starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request Jul 26, 2023
@zy-kkk
[case]else oppo (apache#39)
f3ae47a 
* oppo
morrySnow pushed a commit that referenced this pull request Apr 21, 2024
@seawinde
[improvement](mtmv) Support union rewrite when the materialized view …
06c9fb6 
…is not enough to provide all the data for the query (#33800)

When the materialized view is not enough to provide all the data for the query, if the materialized view is increment update by partition. we can union materialized view and origin query to reponse the query.

this depends on #33362

such as materialized view def is as following:

>         CREATE MATERIALIZED VIEW mv_10086
>         BUILD IMMEDIATE REFRESH AUTO ON MANUAL
>         partition by(l_shipdate)
>         DISTRIBUTED BY RANDOM BUCKETS 2
>         PROPERTIES ('replication_num' = '1') 
>         AS 
>     select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total
>     from lineitem
>     left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate
>     group by
>     l_shipdate,
>     o_orderdate,
>     l_partkey,
>     l_suppkey;

the materialized view data is as following:
+------------+-------------+-----------+-----------+-----------+
| l_shipdate | o_orderdate | l_partkey | l_suppkey | sum_total |
+------------+-------------+-----------+-----------+-----------+
| 2023-10-18 | 2023-10-18  |         2 |         3 |    109.20 |
| 2023-10-17 | 2023-10-17  |         2 |         3 |     99.50 |
| 2023-10-19 | 2023-10-19  |         2 |         3 |     99.50 |
+------------+-------------+-----------+-----------+-----------+

when we insert data to partition `2023-10-17`,  if we run query as following
```
    select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total
    from lineitem
    left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate
    group by
    l_shipdate,
    o_orderdate,
    l_partkey,
    l_suppkey;
```
query rewrite by materialzied view will fail with message   `Check partition query used validation fail`
if we turn on the switch `SET enable_materialized_view_union_rewrite = true;` default true
we run the query above again, it will success and will use union all  materialized view and origin query to response the query correctly. the plan is as following:


```
| Explain String(Nereids Planner)                                                                                                                                                                    |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                                                                                                    |
|   OUTPUT EXPRS:                                                                                                                                                                                    |
|     l_shipdate[#52]                                                                                                                                                                                |
|     o_orderdate[#53]                                                                                                                                                                               |
|     l_partkey[#54]                                                                                                                                                                                 |
|     l_suppkey[#55]                                                                                                                                                                                 |
|     sum_total[#56]                                                                                                                                                                                 |
|   PARTITION: UNPARTITIONED                                                                                                                                                                         |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   VRESULT SINK                                                                                                                                                                                     |
|      MYSQL_PROTOCAL                                                                                                                                                                                |
|                                                                                                                                                                                                    |
|   11:VEXCHANGE                                                                                                                                                                                     |
|      offset: 0                                                                                                                                                                                     |
|      distribute expr lists:                                                                                                                                                                        |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 1                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                   |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 11                                                                                                                                                                                |
|     UNPARTITIONED                                                                                                                                                                                  |
|                                                                                                                                                                                                    |
|   10:VUNION(756)                                                                                                                                                                                   |
|   |                                                                                                                                                                                                |
|   |----9:VAGGREGATE (merge finalize)(753)                                                                                                                                                          |
|   |    |  output: sum(partial_sum(o_totalprice)[#46])[#51]                                                                                                                                         |
|   |    |  group by: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                              |
|   |    |  cardinality=2                                                                                                                                                                            |
|   |    |  distribute expr lists: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                 |
|   |    |                                                                                                                                                                                           |
|   |    8:VEXCHANGE                                                                                                                                                                                 |
|   |       offset: 0                                                                                                                                                                                |
|   |       distribute expr lists: l_shipdate[#42]                                                                                                                                                   |
|   |                                                                                                                                                                                                |
|   1:VEXCHANGE                                                                                                                                                                                      |
|      offset: 0                                                                                                                                                                                     |
|      distribute expr lists:                                                                                                                                                                        |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 2                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25]                                                                                                                                   |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 08                                                                                                                                                                                |
|     HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                            |
|                                                                                                                                                                                                    |
|   7:VAGGREGATE (update serialize)(747)                                                                                                                                                             |
|   |  STREAMING                                                                                                                                                                                     |
|   |  output: partial_sum(o_totalprice[#41])[#46]                                                                                                                                                   |
|   |  group by: l_shipdate[#37], o_orderdate[#38], l_partkey[#39], l_suppkey[#40]                                                                                                                   |
|   |  cardinality=2                                                                                                                                                                                 |
|   |  distribute expr lists: l_shipdate[#37]                                                                                                                                                        |
|   |                                                                                                                                                                                                |
|   6:VHASH JOIN(741)                                                                                                                                                                                |
|   |  join op: RIGHT OUTER JOIN(PARTITIONED)[]                                                                                                                                                      |
|   |  equal join conjunct: (o_orderkey[#21] = l_orderkey[#5])                                                                                                                                       |
|   |  equal join conjunct: (o_orderdate[#25] = l_shipdate[#15])                                                                                                                                     |
|   |  runtime filters: RF000[min_max] <- l_orderkey[#5](2/2/2048), RF001[bloom] <- l_orderkey[#5](2/2/2048), RF002[min_max] <- l_shipdate[#15](1/1/2048), RF003[bloom] <- l_shipdate[#15](1/1/2048) |
|   |  cardinality=2                                                                                                                                                                                 |
|   |  vec output tuple id: 4                                                                                                                                                                        |
|   |  output tuple id: 4                                                                                                                                                                            |
|   |  vIntermediate tuple ids: 3                                                                                                                                                                    |
|   |  hash output slot ids: 6 7 24 25 15                                                                                                                                                            |
|   |  final projections: l_shipdate[#36], o_orderdate[#32], l_partkey[#34], l_suppkey[#35], o_totalprice[#31]                                                                                       |
|   |  final project output tuple id: 4                                                                                                                                                              |
|   |  distribute expr lists: o_orderkey[#21], o_orderdate[#25]                                                                                                                                      |
|   |  distribute expr lists: l_orderkey[#5], l_shipdate[#15]                                                                                                                                        |
|   |                                                                                                                                                                                                |
|   |----3:VEXCHANGE                                                                                                                                                                                 |
|   |       offset: 0                                                                                                                                                                                |
|   |       distribute expr lists: l_orderkey[#5]                                                                                                                                                    |
|   |                                                                                                                                                                                                |
|   5:VEXCHANGE                                                                                                                                                                                      |
|      offset: 0                                                                                                                                                                                     |
|      distribute expr lists:                                                                                                                                                                        |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 3                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: RANDOM                                                                                                                                                                                |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 05                                                                                                                                                                                |
|     HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25]                                                                                                                                            |
|                                                                                                                                                                                                    |
|   4:VOlapScanNode(722)                                                                                                                                                                             |
|      TABLE: union_db.orders(orders), PREAGGREGATION: ON                                                                                                                                            |
|      runtime filters: RF000[min_max] -> o_orderkey[#21], RF001[bloom] -> o_orderkey[#21], RF002[min_max] -> o_orderdate[#25], RF003[bloom] -> o_orderdate[#25]                                     |
|      partitions=3/3 (p_20231017,p_20231018,p_20231019), tablets=9/9, tabletList=161188,161190,161192 ...                                                                                           |
|      cardinality=3, avgRowSize=0.0, numNodes=1                                                                                                                                                     |
|      pushAggOp=NONE                                                                                                                                                                                |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 4                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: HASH_PARTITIONED: l_orderkey[#5]                                                                                                                                                      |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 03                                                                                                                                                                                |
|     HASH_PARTITIONED: l_orderkey[#5], l_shipdate[#15]                                                                                                                                              |
|                                                                                                                                                                                                    |
|   2:VOlapScanNode(729)                                                                                                                                                                             |
|      TABLE: union_db.lineitem(lineitem), PREAGGREGATION: ON                                                                                                                                        |
|      PREDICATES: (l_shipdate[#15] >= '2023-10-17') AND (l_shipdate[#15] < '2023-10-18')                                                                                                            |
|      partitions=1/3 (p_20231017), tablets=3/3, tabletList=161223,161225,161227                                                                                                                     |
|      cardinality=2, avgRowSize=0.0, numNodes=1                                                                                                                                                     |
|      pushAggOp=NONE                                                                                                                                                                                |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 5                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: RANDOM                                                                                                                                                                                |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 01                                                                                                                                                                                |
|     RANDOM                                                                                                                                                                                         |
|                                                                                                                                                                                                    |
|   0:VOlapScanNode(718)                                                                                                                                                                             |
|      TABLE: union_db.mv_10086(mv_10086), PREAGGREGATION: ON                                                                                                                                        |
|      partitions=2/3 (p_20231018_20231019,p_20231019_20231020), tablets=4/4, tabletList=161251,161253,161265 ...                                                                                    |
|      cardinality=2, avgRowSize=0.0, numNodes=1                                                                                                                                                     |
|      pushAggOp=NONE                                                                                                                                                                                |
|                                                                                                                                                                                                    |
| MaterializedView                                                                                                                                                                                   |
| MaterializedViewRewriteSuccessAndChose:                                                                                                                                                            |
|   Names: mv_10086                                                                                                                                                                                  |
| MaterializedViewRewriteSuccessButNotChose:                                                                                                                                                         |
|                                                                                                                                                                                                    |
| MaterializedViewRewriteFail:                                                                                                                                                                       |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
yiguolei pushed a commit that referenced this pull request Apr 21, 2024
@seawinde @Doris-Extras
[improvement](mtmv) Support union rewrite when the materialized view …
8096753 
…is not enough to provide all the data for the query (#33800)

When the materialized view is not enough to provide all the data for the query, if the materialized view is increment update by partition. we can union materialized view and origin query to reponse the query.

this depends on #33362

such as materialized view def is as following:

>         CREATE MATERIALIZED VIEW mv_10086
>         BUILD IMMEDIATE REFRESH AUTO ON MANUAL
>         partition by(l_shipdate)
>         DISTRIBUTED BY RANDOM BUCKETS 2
>         PROPERTIES ('replication_num' = '1') 
>         AS 
>     select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total
>     from lineitem
>     left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate
>     group by
>     l_shipdate,
>     o_orderdate,
>     l_partkey,
>     l_suppkey;

the materialized view data is as following:
+------------+-------------+-----------+-----------+-----------+
| l_shipdate | o_orderdate | l_partkey | l_suppkey | sum_total |
+------------+-------------+-----------+-----------+-----------+
| 2023-10-18 | 2023-10-18  |         2 |         3 |    109.20 |
| 2023-10-17 | 2023-10-17  |         2 |         3 |     99.50 |
| 2023-10-19 | 2023-10-19  |         2 |         3 |     99.50 |
+------------+-------------+-----------+-----------+-----------+

when we insert data to partition `2023-10-17`,  if we run query as following
```
    select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total
    from lineitem
    left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate
    group by
    l_shipdate,
    o_orderdate,
    l_partkey,
    l_suppkey;
```
query rewrite by materialzied view will fail with message   `Check partition query used validation fail`
if we turn on the switch `SET enable_materialized_view_union_rewrite = true;` default true
we run the query above again, it will success and will use union all  materialized view and origin query to response the query correctly. the plan is as following:


```
| Explain String(Nereids Planner)                                                                                                                                                                    |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                                                                                                    |
|   OUTPUT EXPRS:                                                                                                                                                                                    |
|     l_shipdate[#52]                                                                                                                                                                                |
|     o_orderdate[#53]                                                                                                                                                                               |
|     l_partkey[#54]                                                                                                                                                                                 |
|     l_suppkey[#55]                                                                                                                                                                                 |
|     sum_total[#56]                                                                                                                                                                                 |
|   PARTITION: UNPARTITIONED                                                                                                                                                                         |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   VRESULT SINK                                                                                                                                                                                     |
|      MYSQL_PROTOCAL                                                                                                                                                                                |
|                                                                                                                                                                                                    |
|   11:VEXCHANGE                                                                                                                                                                                     |
|      offset: 0                                                                                                                                                                                     |
|      distribute expr lists:                                                                                                                                                                        |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 1                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                   |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 11                                                                                                                                                                                |
|     UNPARTITIONED                                                                                                                                                                                  |
|                                                                                                                                                                                                    |
|   10:VUNION(756)                                                                                                                                                                                   |
|   |                                                                                                                                                                                                |
|   |----9:VAGGREGATE (merge finalize)(753)                                                                                                                                                          |
|   |    |  output: sum(partial_sum(o_totalprice)[#46])[#51]                                                                                                                                         |
|   |    |  group by: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                              |
|   |    |  cardinality=2                                                                                                                                                                            |
|   |    |  distribute expr lists: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                 |
|   |    |                                                                                                                                                                                           |
|   |    8:VEXCHANGE                                                                                                                                                                                 |
|   |       offset: 0                                                                                                                                                                                |
|   |       distribute expr lists: l_shipdate[#42]                                                                                                                                                   |
|   |                                                                                                                                                                                                |
|   1:VEXCHANGE                                                                                                                                                                                      |
|      offset: 0                                                                                                                                                                                     |
|      distribute expr lists:                                                                                                                                                                        |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 2                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25]                                                                                                                                   |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 08                                                                                                                                                                                |
|     HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45]                                                                                                            |
|                                                                                                                                                                                                    |
|   7:VAGGREGATE (update serialize)(747)                                                                                                                                                             |
|   |  STREAMING                                                                                                                                                                                     |
|   |  output: partial_sum(o_totalprice[#41])[#46]                                                                                                                                                   |
|   |  group by: l_shipdate[#37], o_orderdate[#38], l_partkey[#39], l_suppkey[#40]                                                                                                                   |
|   |  cardinality=2                                                                                                                                                                                 |
|   |  distribute expr lists: l_shipdate[#37]                                                                                                                                                        |
|   |                                                                                                                                                                                                |
|   6:VHASH JOIN(741)                                                                                                                                                                                |
|   |  join op: RIGHT OUTER JOIN(PARTITIONED)[]                                                                                                                                                      |
|   |  equal join conjunct: (o_orderkey[#21] = l_orderkey[#5])                                                                                                                                       |
|   |  equal join conjunct: (o_orderdate[#25] = l_shipdate[#15])                                                                                                                                     |
|   |  runtime filters: RF000[min_max] <- l_orderkey[#5](2/2/2048), RF001[bloom] <- l_orderkey[#5](2/2/2048), RF002[min_max] <- l_shipdate[#15](1/1/2048), RF003[bloom] <- l_shipdate[#15](1/1/2048) |
|   |  cardinality=2                                                                                                                                                                                 |
|   |  vec output tuple id: 4                                                                                                                                                                        |
|   |  output tuple id: 4                                                                                                                                                                            |
|   |  vIntermediate tuple ids: 3                                                                                                                                                                    |
|   |  hash output slot ids: 6 7 24 25 15                                                                                                                                                            |
|   |  final projections: l_shipdate[#36], o_orderdate[#32], l_partkey[#34], l_suppkey[#35], o_totalprice[#31]                                                                                       |
|   |  final project output tuple id: 4                                                                                                                                                              |
|   |  distribute expr lists: o_orderkey[#21], o_orderdate[#25]                                                                                                                                      |
|   |  distribute expr lists: l_orderkey[#5], l_shipdate[#15]                                                                                                                                        |
|   |                                                                                                                                                                                                |
|   |----3:VEXCHANGE                                                                                                                                                                                 |
|   |       offset: 0                                                                                                                                                                                |
|   |       distribute expr lists: l_orderkey[#5]                                                                                                                                                    |
|   |                                                                                                                                                                                                |
|   5:VEXCHANGE                                                                                                                                                                                      |
|      offset: 0                                                                                                                                                                                     |
|      distribute expr lists:                                                                                                                                                                        |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 3                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: RANDOM                                                                                                                                                                                |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 05                                                                                                                                                                                |
|     HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25]                                                                                                                                            |
|                                                                                                                                                                                                    |
|   4:VOlapScanNode(722)                                                                                                                                                                             |
|      TABLE: union_db.orders(orders), PREAGGREGATION: ON                                                                                                                                            |
|      runtime filters: RF000[min_max] -> o_orderkey[#21], RF001[bloom] -> o_orderkey[#21], RF002[min_max] -> o_orderdate[#25], RF003[bloom] -> o_orderdate[#25]                                     |
|      partitions=3/3 (p_20231017,p_20231018,p_20231019), tablets=9/9, tabletList=161188,161190,161192 ...                                                                                           |
|      cardinality=3, avgRowSize=0.0, numNodes=1                                                                                                                                                     |
|      pushAggOp=NONE                                                                                                                                                                                |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 4                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: HASH_PARTITIONED: l_orderkey[#5]                                                                                                                                                      |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 03                                                                                                                                                                                |
|     HASH_PARTITIONED: l_orderkey[#5], l_shipdate[#15]                                                                                                                                              |
|                                                                                                                                                                                                    |
|   2:VOlapScanNode(729)                                                                                                                                                                             |
|      TABLE: union_db.lineitem(lineitem), PREAGGREGATION: ON                                                                                                                                        |
|      PREDICATES: (l_shipdate[#15] >= '2023-10-17') AND (l_shipdate[#15] < '2023-10-18')                                                                                                            |
|      partitions=1/3 (p_20231017), tablets=3/3, tabletList=161223,161225,161227                                                                                                                     |
|      cardinality=2, avgRowSize=0.0, numNodes=1                                                                                                                                                     |
|      pushAggOp=NONE                                                                                                                                                                                |
|                                                                                                                                                                                                    |
| PLAN FRAGMENT 5                                                                                                                                                                                    |
|                                                                                                                                                                                                    |
|   PARTITION: RANDOM                                                                                                                                                                                |
|                                                                                                                                                                                                    |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                        |
|                                                                                                                                                                                                    |
|   STREAM DATA SINK                                                                                                                                                                                 |
|     EXCHANGE ID: 01                                                                                                                                                                                |
|     RANDOM                                                                                                                                                                                         |
|                                                                                                                                                                                                    |
|   0:VOlapScanNode(718)                                                                                                                                                                             |
|      TABLE: union_db.mv_10086(mv_10086), PREAGGREGATION: ON                                                                                                                                        |
|      partitions=2/3 (p_20231018_20231019,p_20231019_20231020), tablets=4/4, tabletList=161251,161253,161265 ...                                                                                    |
|      cardinality=2, avgRowSize=0.0, numNodes=1                                                                                                                                                     |
|      pushAggOp=NONE                                                                                                                                                                                |
|                                                                                                                                                                                                    |
| MaterializedView                                                                                                                                                                                   |
| MaterializedViewRewriteSuccessAndChose:                                                                                                                                                            |
|   Names: mv_10086                                                                                                                                                                                  |
| MaterializedViewRewriteSuccessButNotChose:                                                                                                                                                         |
|                                                                                                                                                                                                    |
| MaterializedViewRewriteFail:                                                                                                                                                                       |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
@yx-keith yx-keith mentioned this pull request May 10, 2024
Closed
3 tasks
yiguolei pushed a commit that referenced this pull request Aug 10, 2024
@mrhhsg
[fix](ub) undefined behavior in FixedContainer (#39191)
7697f02 
## Proposed changes

Undefined behavior occurs if there is a null value in the list.

```
/root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null
/var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here
#0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41
#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60
#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36
#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63
#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21
#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26
#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19
#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5
#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9
#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12
#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38
#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5
#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9
#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5
#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5
#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25
#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9
#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19
#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33
#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29
#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h
#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24
#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14
#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27
#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9
#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5
#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19
#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32
#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8
#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in
```
mrhhsg added a commit to mrhhsg/doris that referenced this pull request Aug 11, 2024
@mrhhsg
[fix](ub) undefined behavior in FixedContainer (apache#39191)
2124dae 
## Proposed changes

Undefined behavior occurs if there is a null value in the list.

```
/root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null
/var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here
#0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41
apache#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60
apache#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36
apache#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63
apache#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21
apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26
apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19
apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5
apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9
apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12
apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38
apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5
apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9
apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5
apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5
apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25
apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9
apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19
apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33
apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29
apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h
apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24
apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14
apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27
apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9
apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5
apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19
apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32
apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8
apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in
```
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@mrhhsg @dataroaring
[fix](ub) undefined behavior in FixedContainer (#39191)
a15ccb0 
## Proposed changes

Undefined behavior occurs if there is a null value in the list.

```
/root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null
/var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here
#0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41
#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60
#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36
#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63
#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21
#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26
#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19
#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5
#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9
#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12
#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38
#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5
#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9
#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5
#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5
#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25
#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9
#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19
#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33
#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29
#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h
#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24
#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14
#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27
#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9
#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5
#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19
#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32
#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8
#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in
```
yiguolei pushed a commit that referenced this pull request Aug 12, 2024
@mrhhsg
[fix](ub) undefined behavior in FixedContainer (#39191) (#39201)
96260d9 
## Proposed changes

pick #39191
Undefined behavior occurs if there is a null value in the list.

```
/root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null
/var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here
#0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41
#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60
#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36
#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63
#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21
#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26
#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19
#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5
#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9
#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12
#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38
#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5
#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9
#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5
#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5
#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25
#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9
#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19
#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33
#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29
#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h
#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24
#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14
#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27
#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9
#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5
#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19
#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32
#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8
#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@mrhhsg @wyxxxcat
[fix](ub) undefined behavior in FixedContainer (apache#39191)
e93604e 
## Proposed changes

Undefined behavior occurs if there is a null value in the list.

```
/root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null
/var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here
#0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41
#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60
apache#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36
apache#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63
apache#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21
apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26
apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19
apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5
apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9
apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12
apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38
apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5
apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9
apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5
apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5
apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25
apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9
apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19
apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33
apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29
apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h
apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24
apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14
apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27
apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9
apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5
apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19
apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32
apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8
apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in
```
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@mrhhsg @dataroaring
[fix](ub) undefined behavior in FixedContainer (#39191)
843cc86 
## Proposed changes

Undefined behavior occurs if there is a null value in the list.

```
/root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null
/var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here
#0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41
#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60
#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36
#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63
#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21
#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26
#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19
#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5
#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9
#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12
#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38
#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5
#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9
#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5
#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5
#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25
#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9
#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19
#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33
#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29
#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h
#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24
#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14
#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27
#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9
#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5
#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19
#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32
#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8
#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in
```
mrhhsg added a commit to mrhhsg/doris that referenced this pull request Sep 21, 2024
@mrhhsg
[fix](ub) undefined behavior in FixedContainer (apache#39191)
21ba579 
## Proposed changes

Undefined behavior occurs if there is a null value in the list.

```
/root/doris/be/src/vec/common/string_ref.h:271:54: runtime error: null pointer passed as argument 2, which is declared to never be null
/var/local/ldb-toolchain/bin/../usr/include/string.h:64:33: note: nonnull attribute specified here
#0 0x5616d072245d in doris::StringRef::eq(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:271:41
apache#1 0x5616d072245d in doris::StringRef::operator==(doris::StringRef const&) const /root/doris/be/src/vec/common/string_ref.h:274:60
apache#2 0x5616d072245d in doris::FixedContainer::find(doris::StringRef const&) const /root/doris/be/src/exprs/hybrid_set.h:76:36
apache#3 0x5616d072245d in void doris::StringValueSet>::_find_batch(doris::vectorized::IColumn const&, unsigned long, doris::vectorized::PODArray, 16ul, 15ul> const*, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/exprs/hybrid_set.h:688:63
apache#4 0x5616d0747857 in doris::vectorized::FunctionIn::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/in.h:170:21
apache#5 0x5616c741fa3a in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:462:26
apache#6 0x5616cbb5b650 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
apache#7 0x5616cbb4e14e in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
apache#8 0x5616cbb4e3c2 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
apache#9 0x5616c741cd68 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector> const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:190:19
apache#10 0x5616c74cf712 in doris::vectorized::VInPredicate::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vin_predicate.cpp:130:5
apache#11 0x5616c740d5c0 in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:183:9
apache#12 0x5616c740ecf5 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:215:12
apache#13 0x5616c7462e24 in doris::vectorized::VCompoundPred::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vcompound_pred.h:127:38
apache#14 0x5616c74bccec in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:54:5
apache#15 0x5616c74c1dcc in doris::vectorized::VExprContext::execute_conjuncts(std::vector, std::allocator>> const&, std::vector, 16ul, 15ul>, std::allocator, 16ul, 15ul>>> const*, bool, doris::vectorized::Block*, doris::vectorized::PODArray, 16ul, 15ul>, bool) /root/doris/be/src/vec/exprs/vexpr_context.cpp:169:9
apache#16 0x5616c74c5108 in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block(std::vector, std::allocator>> const&, doris::vectorized::Block*, std::vector>&, int, doris::vectorized::PODArray, 16ul, 15ul>&) /root/doris/be/src/vec/exprs/vexpr_context.cpp:322:5
apache#17 0x5616ad8a7f1a in doris::segment_v2::SegmentIterator::_execute_common_expr(unsigned short*, unsigned short&, doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2680:5
apache#18 0x5616ad89e86e in doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2582:25
apache#19 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2315:9
apache#20 0x5616ad892f5c in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2314:19
apache#21 0x5616ad6dd9cc in doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44:33
apache#22 0x5616ad269d67 in doris::BetaRowsetReader::next_block(doris::vectorized::Block*) /root/doris/be/src/olap/rowset/beta_rowset_reader.cpp:380:29
apache#23 0x5616de6de110 in doris::vectorized::VCollectIterator::Level0Iterator::_refresh() /root/doris/be/src/vec/olap/vcollect_iterator.h
apache#24 0x5616de6c967f in doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:514:24
apache#25 0x5616de6ca8a6 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:493:14
apache#26 0x5616de6d7008 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() /root/doris/be/src/vec/olap/vcollect_iterator.cpp:692:27
apache#27 0x5616de6bd200 in doris::vectorized::VCollectIterator::build_heap(std::vector, std::allocator>>&) /root/doris/be/src/vec/olap/vcollect_iterator.cpp:186:9
apache#28 0x5616de651b6c in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:157:5
apache#29 0x5616de65526f in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /root/doris/be/src/vec/olap/block_reader.cpp:229:19
apache#30 0x5616e175a0f9 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:237:32
apache#31 0x5616c736ad34 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
apache#32 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
apache#33 0x5616c736f05e in doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
apache#34 0x5616c736f05e in void std::_invoke_impl, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(std::_invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
apache#35 0x5616c736f05e in std::enable_if, std::shared_ptr)::$1::operator()() const::'lambda'()&>, void>::type std::_invoke_r, std::shared_ptr)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
apache#36 0x5616c736f05e in std::_Function_handler, std::shared_ptr)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
apache#37 0x5616aeed6a3b in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
apache#38 0x5616aeeae4f7 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
apache#39 0x7f7e663e3ac2 in start_thread nptl/pthread_create.c:442:8
apache#40 0x7f7e6647584f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /root/doris/be/src/vec/common/string_ref.h:271:54 in
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@chenhao7253886